Exposing pixel groups in producing digital images

ABSTRACT

An imaging system and method for producing a digital image from pixel signals captured by a pixel array is disclosed. First pixel signals, generated during a first exposure period, are read out from a first group of pixels of the pixel array. The first group of pixels is reset after reading out the first pixel signals. Second pixel signals from the first group of pixel are read out after resetting the first group of pixels. The second pixel signals are generated during a second exposure period. Third pixel signals from a second group of pixels of the pixel array are read out. The third pixel signals are generated during a third exposure period that overlaps at least a portion of the first and second exposure periods. The first, second, and third pixel signals are used to produce the digital image.

CROSS-REFERENCE

This application is a continuation of, and claims priority to, U.S.application Ser. No. 12/416,172, filed Apr. 1, 2009, now pending.

Reference is made to commonly-assigned U.S. patent application Ser. No.11/780,523 filed Jul. 20, 2007 entitled “MULTIPLE COMPONENT READ OUT OFIMAGE SENSOR” to Hamilton, et al., the disclosure of which isincorporated herein.

FIELD OF THE INVENTION

The present invention relates to an image capture device using atwo-dimensional image sensor array with multiple exposures and readoutsto produce a digital image.

BACKGROUND OF THE INVENTION

In digital imaging, it is desirable to capture an image sequence havinghigh image quality, high spatial resolution and high temporalresolution, also referred to as frame rate. With many current imagesequence capture devices, however, it is not possible to obtain suchhigh quality image sequences. In many cases, one of the desired imagesequence attributes is obtained at the expense of the others. Forexample, in a conventional image sequence capture device, the exposureduration for a given image is limited by the frame rate. The higher theframe rate, the shorter each image exposure must be. In a low-lightenvironment, individual image captures within an image sequence canreceive insufficient light and produce noisy images. The quality of agiven image with respect to noise can be improved by utilizing longerexposure durations for each image, but this comes at the expense of alower frame rate. Alternatively, the image quality with respect to noisecan be improved by combining pixels through the technique of binning;however this improvement comes at the expense of lower spatialresolution. In many cases the spatial and temporal resolution of theimage sequence are limited by the readout capabilities of the sensor. Asensor is capable of reading a certain number of pixels per second. Thisreadout capability is balanced between spatial and temporal resolutionof the readouts. Increasing one must come at the expense of the other inorder to keep the total number of pixels read within the achievablerange of the sensor.

Many solutions have been proposed to allow a digital image sequencecapture device to capture image sequences with improved quality andresolution. One method to reduce noise in a digital image sequence isthrough temporal noise cleaning. An example of such a technique is givenin U.S. Pat. No. 7,330,218. Temporal noise reduction techniques exploitthe high temporal correlation among neighboring images to achieve noisereduction. In static scenes, multiple readouts of the same image scenecontent are available in successive images, allowing for effective noisereduction. The drawbacks of temporal noise reduction include memoryrequirements to buffer multiple images, as well as computationalrequirements to filter the images, in particular if motion estimationand compensation are used to align regions of local or global motion.Additionally, temporal noise reduction does nothing to improve thespatial or temporal resolution of the image sequence.

One method to improve temporal resolution is temporal frameinterpolation. Those skilled in the art will recognize, however, thatsuch techniques are computationally complex, memory intensive, and oftengenerate artifacts in the interpolated frames.

One method to improve the spatial resolution of an image sequence isthrough super-resolution techniques. Examples of super-resolutionalgorithms are provided in U.S. Pat. Nos. 7,215,831 and 7,379,612. Videosuper-resolution techniques use neighboring frames to estimate eachhigh-resolution video frame. The drawbacks of spatial videosuper-resolution include computational complexity and memoryrequirements. Dynamic scenes are also difficult for spatialsuper-resolution algorithms to process.

Another method to improve the quality of a digital image sequence isthrough the use of a dual-sensor camera. Such a system is proposed in USPatent Application 2008/0211941, “Digital Camera Using Multiple ImageSensors to Provide Improved Temporal Sampling.” Improved temporalresolution can be achieved by staggering the exposures of the dualsensors Improved image quality and noise reduction are possible byexposing the two sensors equally and then combining the resultantimages. The drawbacks to this solution include the costs associated witha dual sensor camera. Additionally, in a dual-lens device, the needexists to spatially align images captured from the different lenssystems.

Another method to improve spatial resolution is by capturing anintermittent high resolution image along with the low resolution imagesequence, followed by processing to generate an entire high resolutionimage sequence from the aggregate data. Examples of such solutions areU.S. Pat. Nos. 7,110,025 and 7,372,504. The drawbacks of this solutioninclude in some cases the requirement of an additional sensor and otherhardware to capture the high resolution image without disrupting theimage sequence capture process. Other drawbacks include the need tobuffer multiple images, depending on the frequency and usage of the highresolution images in generating the final high resolution imagesequence.

Another method for improving the quality of an image sequence is throughthe use of an image sensor with improved light sensitivity. Many imagesensors use a combination of red, green and blue color filters arrangedin the familiar Bayer pattern, as described in U.S. Pat. No. 3,971,065.As solutions for improving image capture under varying light conditionsand for improving overall sensitivity of the imaging sensor,modifications to the familiar Bayer pattern have been disclosed. Forexample, commonly assigned U.S. Patent Applications Publication No.2007/0046807 entitled “Capturing Images Under Varying LightingConditions” by Hamilton et al. and Publication No. 2007/0024931 entitled“Image Sensor with Improved Light Sensitivity” by Compton et al. bothdescribe alternative sensor arrangements that combine color filters withpanchromatic filter elements, interleaved in some manner. With this typeof solution, some portion of the image sensor detects color; the otherpanchromatic portion is optimized to detect light spanning the visibleband for improved dynamic range and sensitivity. These solutions thusprovide a pattern of pixels, some pixels with color filters (providing anarrow-band spectral response) and some without (unfiltered pixels orpixels filtered to provide a broad-band spectral response).

Using a combination of both narrow- and wide-spectral band pixelresponses, image sensors can be used at lower light levels or provideshorter exposure durations. See Sato et al in U.S. Pat. No. 4,390,895,Yamagami et al in U.S. Pat. No. 5,323,233, and Gindele et al in U.S.Pat. No. 6,476,865. Such sensors can provide improved image quality atlow light levels, but additional techniques are required to address theneed for producing image sequences with improved spatial and temporalresolution.

In digital imaging, it is also desirable to capture an image sequencehaving high dynamic range. In photography and imaging, the dynamic rangerepresents the ratio of two luminance values, with the luminanceexpressed in candelas per square meter. The range of luminance humanvision can handle is quite large. While the luminance of starlight isaround 0.001 cd/m2, that of a sunlit scene is around 100,000 cd/m2,which is one hundred million times higher. The luminance of the sunitself is approximately 1,000,000,000 cd/m2. The human eye canaccommodate a dynamic range of approximately 10,000:1 in a single view.The dynamic range for a camera is defined as the ratio of the intensitythat just saturates the camera to the intensity that just lifts thecamera response one standard deviation above camera noise. In mostcommercially available sensors today, the maximum ratio of signal tonoise for a pixel is about 100:1. This, in turn, represents the maximumdynamic range of the pixel.

Since most digital cameras are only able to capture a limited dynamicrange (the exposure setting determines which part of the total dynamicrange will be captured), high dynamic range images are commonly createdfrom captures of the same scene taken under different exposure levels.For most daylight outdoor scenes excluding the sun, three exposuresspaced by two exposure values apart are often sufficient to properlycover the dynamic range. However, this method requires a scene that doesnot change between the captures in the series.

Jones (U.S. Pat. No. 6,924,841 B2) discloses a method for extending thedynamic range of a sensor by having two groups of pixels with differentsensitivities. However, Jones requires that the sensitivity of the firstgroup of pixels overlaps with the sensitivity of the second group ofpixels in order to have some common dynamic range. This method is notdesirable because it will not provide a substantial dynamic range forreal world scenes. It also requires a specialized sensor with pixels ofdifferent sensitivities.

Kindt et al. in U.S. Pat. No. 6,348,681 discloses a method and circuitfor setting breakpoints for a sensor to achieve a user selectedpiecewise linear transfer function.

Ando et al. in U.S. Pat. No. 7,446,812 discloses a method for using dualintegration periods during a same frame and readout to increase thedynamic range for a capture. This method does not utilize every photonthat reaches the sensor because the pixels with shorter integration timewill not capture photons between the time of the readout of those pixelsand the pixels with the longer integration time.

Thus, there exists a need for producing a digital image sequence withimproved image quality, spatial resolution and temporal resolution,without generating spatial or temporal artifacts, and withoutsignificant memory costs, computational costs, or hardware costs.

There also exists a need for producing a high dynamic range image froman image sensor without fundamentally increasing the complexity orcomposition of the individual pixels in the sensor.

SUMMARY OF THE INVENTION

An advantage of an embodiment of the invention is that sequences ofcolor images with increased spatial resolution, temporal resolution andimage quality can be produced without the need for additional lenses andimage sensor arrays.

A further advantage of an embodiment of the invention is that sequencesof color images with increased spatial resolution, temporal resolutionand image quality can be produced without the need for computationallycomplex and memory intensive algorithms.

A further advantage of an embodiment of the invention is thatcombinations of sequences of low spatial resolution, high temporalresolution, color images and sequences of high spatial resolution, lowtemporal resolution, color images can be produced without the need foradditional lenses and image sensor arrays.

A further advantage of an embodiment of the invention is that theextended dynamic range image can be produced without the need foradditional lenses and image sensor arrays.

A further advantage of an embodiment of the invention is that theextended dynamic range image can be produced with lower buffering andwithout the need for computationally complex and memory intensivealgorithms.

This and other aspects, objects, features, and advantages of the presentinvention will be more clearly understood and appreciated from a reviewof the following detailed description of the preferred embodiments andappended claims, and by reference to the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a conventional digital still camera systemthat can employ a conventional sensor and processing methods or thesensor and processing methods of the current invention;

FIG. 2 (prior art) is conventional Bayer color filter array patternshowing a minimal repeating unit and a non-minimal repeating unit;

FIGS. 3A and 3B (prior art) show timing diagrams for rolling shutteroperation under various light conditions;

FIG. 4 (prior art) provides representative spectral quantum efficiencycurves for red, green, and blue pixels, as well as a wider spectrumpanchromatic quantum efficiency, all multiplied by the transmissioncharacteristics of an infrared cut filter;

FIG. 5 illustrates a timing diagram for an embodiment of the currentinvention;

FIG. 6 is a flow diagram showing an embodiment of the current invention;

FIG. 7 (prior art) is a diagram showing an example color filter arraypattern containing panchromatic and color pixels;

FIG. 8 (prior art) is a schematic diagram showing how pixels in adjacentrows can be binned together, sharing the same floating diffusioncomponent;

FIG. 9 is a timing diagram showing rolling shutter operation forpanchromatic and color pixels in one embodiment of the currentinvention;

FIG. 10 is a diagram showing the generation of digital images atapproximately the spatial resolution of the sensor from readouts ofpanchromatic pixels and from readouts of binned panchromatic and binnedcolor pixels;

FIG. 11 is a diagram showing the generation of digital images atapproximately one half the horizontal and one half the vertical spatialresolution of the sensor from readouts of panchromatic pixels and fromreadouts of binned panchromatic and binned color pixels;

FIG. 12 is a diagram showing the generation of digital images atapproximately one half the horizontal and one half the vertical spatialresolution of the sensor as well as digital images at approximately thespatial resolution of the sensor from readouts of panchromatic pixelsand from readouts of binned panchromatic and binned color pixels;

FIG. 13 is a flow diagram showing the generation of a residual image;

FIG. 14 is a diagram showing the generation of digital images atapproximately one half the horizontal and one half the vertical spatialresolution of the sensor from readouts of binned panchromatic pixels andfrom readouts of panchromatic and color pixels binned together;

FIG. 15 is a diagram showing an example color filter array patterncontaining panchromatic pixels;

FIG. 16 is a diagram showing the generation of digital images fromreadouts of panchromatic pixels;

FIG. 17 is a diagram showing an example color filter array patterncontaining panchromatic pixels;

FIG. 18 is a diagram showing the generation of digital images forpanchromatic pixels in an extended dynamic range embodiment of thecurrent invention;

FIG. 19 is a diagram showing an example color filter array patterncontaining panchromatic pixels and color pixels; and

FIG. 20 is a timing diagram showing rolling shutter operation forpanchromatic pixels in an embodiment of the current invention.

DETAILED DESCRIPTION OF THE INVENTION

Because digital cameras employing imaging devices and related circuitryfor signal capture and correction and for exposure control are wellknown, the present description will be directed in particular toelements forming part of, or cooperating more directly with, method andapparatus in accordance with the present invention. Elements notspecifically shown or described herein are selected from those known inthe art. Certain aspects of the embodiments to be described are providedin software. Given the system as shown and described according to theinvention in the following materials, software not specifically shown,described or suggested herein that is useful for implementation of theinvention is conventional and within the ordinary skill in such arts.

Turning now to FIG. 1, a block diagram of an image capture device shownas a digital camera embodying the present invention is shown. Although adigital camera will now be explained, the present invention is clearlyapplicable to other types of image capture devices, such as imagingsub-systems included in non-camera devices such as mobile phones andautomotive vehicles, for example. Light 10 from the subject scene isinput to an imaging stage 11, where the light is focused by lens 12 toform an image on solid-state image sensor 20. Image sensor 20 convertsthe incident light to an electrical signal by integrating charge foreach picture element (pixel). The image sensor 20 of the preferredembodiment is a charge coupled device (CCD) type or an active pixelsensor (APS) type. (APS devices are often referred to as CMOS sensorsbecause of the ability to fabricate them in a Complementary Metal OxideSemiconductor process). The sensor includes an arrangement of colorfilters, as described in more detail subsequently. The amount of lightreaching the sensor 20 is regulated by an iris block 14 that varies theaperture and the neutral density (ND) filter block 13 that includes oneor more ND filters interposed in the optical path. Also regulating theoverall light level is the time that the shutter block 18 is open. Theexposure controller block 40 responds to the amount of light availablein the scene as metered by the brightness sensor block 16 and controlsall three of these regulating functions.

The analog signal from image sensor 20 is processed by analog signalprocessor 22 and applied to analog to digital (A/D) converter 24 fordigitizing the sensor signals. Timing generator 26 produces variousclocking signals to select rows and pixels and synchronizes theoperation of analog signal processor 22 and A/D converter 24. The imagesensor stage 28 includes the image sensor 20, the analog signalprocessor 22, the A/D converter 24, and the timing generator 26. Thefunctional elements of image sensor stage 28 are separately fabricatedintegrated circuits, or they are fabricated as a single integratedcircuit as is commonly done with CMOS image sensors. The resultingstream of digital pixel values from A/D converter 24 is stored in memory32 associated with digital signal processor (DSP) 36.

Digital signal processor 36 is one of three processors or controllers inthis embodiment, in addition to system controller 50 and exposurecontroller 40. Although this distribution of camera functional controlamong multiple controllers and processors is typical, these controllersor processors are combined in various ways without affecting thefunctional operation of the camera and the application of the presentinvention. These controllers or processors can comprise one or moredigital signal processor devices, microcontrollers, programmable logicdevices, or other digital logic circuits. Although a combination of suchcontrollers or processors has been described, it should be apparent thatone controller or processor is designated to perform all of the neededfunctions. All of these variations can perform the same function andfall within the scope of this invention, and the term “processing stage”will be used as needed to encompass all of this functionality within onephrase, for example, as in processing stage 38 in FIG. 1.

In the illustrated embodiment, DSP 36 manipulates the digital image datain its memory 32 according to a software program permanently stored inprogram memory 54 and copied to memory 32 for execution during imagecapture. DSP 36 executes the software needed for practicing imageprocessing shown in FIG. 18. Memory 32 includes any type of randomaccess memory, such as SDRAM. A bus 30 comprising a pathway for addressand data signals connects DSP 36 to its related memory 32, A/D converter24 and other related devices.

System controller 50 controls the overall operation of the camera basedon a software program stored in program memory 54, which can includeFlash EEPROM or other nonvolatile memory. This memory can also be usedto store image sensor calibration data, user setting selections andother data which must be preserved when the camera is turned off. Systemcontroller 50 controls the sequence of image capture by directingexposure controller 40 to operate the lens 12, ND filter 13, iris 14,and shutter 18 as previously described, directing the timing generator26 to operate the image sensor 20 and associated elements, and directingDSP 36 to process the captured image data. After an image is capturedand processed, the final image file stored in memory 32 is transferredto a host computer via interface 57, stored on a removable memory card64 or other storage device, and displayed for the user on image display88.

A bus 52 includes a pathway for address, data and control signals, andconnects system controller 50 to DSP 36, program memory 54, systemmemory 56, host interface 57, memory card interface 60 and other relateddevices. Host interface 57 provides a high-speed connection to apersonal computer (PC) or other host computer for transfer of image datafor display, storage, manipulation or printing. This interface is anIEEE1394 or USB2.0 serial interface or any other suitable digitalinterface. Memory card 64 is typically a Compact Flash (CF) cardinserted into socket 62 and connected to the system controller 50 viamemory card interface 60. Other types of storage that are utilizedinclude without limitation PC-Cards, MultiMedia Cards (MMC), or SecureDigital (SD) cards.

Processed images are copied to a display buffer in system memory 56 andcontinuously read out via video encoder 80 to produce a video signal.This signal is output directly from the camera for display on anexternal monitor, or processed by display controller 82 and presented onimage display 88. This display is typically an active matrix colorliquid crystal display (LCD), although other types of displays are usedas well.

The user interface 68, including all or any combination of viewfinderdisplay 70, exposure display 72, status display 76 and image display 88,and user inputs 74, is controlled by a combination of software programsexecuted on exposure controller 40 and system controller 50. User inputs74 typically include some combination of buttons, rocker switches,joysticks, rotary dials or touch screens. Exposure controller 40operates light metering, exposure mode, autofocus and other exposurefunctions. The system controller 50 manages the graphical user interface(GUI) presented on one or more of the displays, e.g., on image display88. The GUI typically includes menus for making various optionselections and review modes for examining captured images.

Exposure controller 40 accepts user inputs selecting exposure mode, lensaperture, exposure time (shutter speed), and exposure index or ISO speedrating and directs the lens and shutter accordingly for subsequentcaptures. Brightness sensor 16 is employed to measure the brightness ofthe scene and provide an exposure meter function for the user to referto when manually setting the ISO speed rating, aperture and shutterspeed. In this case, as the user changes one or more settings, the lightmeter indicator presented on viewfinder display 70 tells the user towhat degree the image will be over or underexposed. In an automaticexposure mode, the user changes one setting and the exposure controller40 automatically alters another setting to maintain correct exposure,e.g., for a given ISO speed rating when the user reduces the lensaperture, the exposure controller 40 automatically increases theexposure time to maintain the same overall exposure.

The ISO speed rating is an important attribute of a digital stillcamera. The exposure time, the lens aperture, the lens transmittance,the level and spectral distribution of the scene illumination, and thescene reflectance determine the exposure level of a digital stillcamera. When an image from a digital still camera is obtained using aninsufficient exposure, proper tone reproduction can generally bemaintained by increasing the electronic or digital gain, but the imagewill contain an unacceptable amount of noise. As the exposure isincreased, the gain is decreased, and therefore the image noise cannormally be reduced to an acceptable level. If the exposure is increasedexcessively, the resulting signal in bright areas of the image canexceed the maximum signal level capacity of the image sensor or camerasignal processing. This can cause image highlights to be clipped to forma uniformly bright area, or to bloom into surrounding areas of theimage. It is important to guide the user in setting proper exposures. AnISO speed rating is intended to serve as such a guide. In order to beeasily understood by photographers, the ISO speed rating for a digitalstill camera should directly relate to the ISO speed rating forphotographic film cameras. For example, if a digital still camera has anISO speed rating of ISO 200, then the same exposure time and apertureshould be appropriate for an ISO 200 rated film/process system.

The ISO speed ratings are intended to harmonize with film ISO speedratings. However, there are differences between electronic andfilm-based imaging systems that preclude exact equivalency. Digitalstill cameras can include variable gain, and can provide digitalprocessing after the image data has been captured, enabling tonereproduction to be achieved over a range of camera exposures. Because ofthis flexibility, digital still cameras can have a range of speedratings. This range is defined as the ISO speed latitude. To preventconfusion, a single value is designated as the inherent ISO speedrating, with the ISO speed latitude upper and lower limits indicatingthe speed range, that is, a range including effective speed ratings thatdiffer from the inherent ISO speed rating. With this in mind, theinherent ISO speed is a numerical value calculated from the exposureprovided at the focal plane of a digital still camera to producespecified camera output signal characteristics. The inherent speed isusually the exposure index value that produces peak image quality for agiven camera system for normal scenes, where the exposure index is anumerical value that is inversely proportional to the exposure providedto the image sensor.

The foregoing description of a digital camera will be familiar to oneskilled in the art. It will be obvious that there are many variations ofthis embodiment that can be selected to reduce the cost, add features,or improve the performance of the camera. For example, an autofocussystem is added, or the lens is detachable and interchangeable. It willbe understood that the present invention is applied to any type ofdigital camera or, more generally, digital image capture apparatus,where alternative modules provide similar functionality.

Given the illustrative example of FIG. 1, the following description willthen describe in detail the operation of this camera for capturing animage sequence according to the present invention. Whenever generalreference is made to an image sensor in the following description, it isunderstood to be representative of the image sensor 20 from FIG. 1.Image sensor 20 shown in FIG. 1 typically includes a two-dimensionalarray of light sensitive pixels fabricated on a silicon substrate thatconvert incoming light at each pixel into an electrical signal that ismeasured. In the context of an image sensor, a pixel (a contraction of“picture element”) refers to a discrete light sensing area and chargeshifting or charge measurement circuitry associated with the lightsensing area. In the context of a digital color image, the term pixelcommonly refers to a particular location in the image having associatedcolor values. The term color pixel will refer to a pixel having a colorphotoresponse over a relatively narrow spectral band. The terms exposureduration and exposure time are used interchangeably.

As sensor 20 is exposed to light, free electrons are generated andcaptured within the electronic structure at each pixel. Capturing thesefree electrons for some period of time and then measuring the number ofelectrons captured, or measuring the rate at which free electrons aregenerated, can measure the light level at each pixel. In the formercase, accumulated charge is shifted out of the array of pixels to acharge-to-voltage measurement circuit as in a charge-coupled device(CCD), or the area close to each pixel can contain elements of acharge-to-voltage measurement circuit as in an active pixel sensor (APSor CMOS sensor).

In order to produce a color image, the array of pixels in an imagesensor typically has a pattern of color filters placed over them. FIG. 2shows a pattern of red (R), green (G), and blue (B) color filters thatis commonly used. This particular pattern is commonly known as a Bayercolor filter array (CFA) after its inventor Bryce Bayer as disclosed inU.S. Pat. No. 3,971,065. This pattern is effectively used in imagesensors having a two-dimensional array of color pixels. As a result,each pixel has a particular color photoresponse that, in this case, is apredominant sensitivity to red, green or blue light. Another usefulvariety of color photoresponses is a predominant sensitivity to magenta,yellow, or cyan light. In each case, the particular color photoresponsehas high sensitivity to certain portions of the visible spectrum, whilesimultaneously having low sensitivity to other portions of the visiblespectrum.

A minimal repeating unit is a repeating unit such that no otherrepeating unit has fewer pixels. For example, the CFA in FIG. 2 includesa minimal repeating unit that is two pixels by two pixels as shown bypixel block 100 in FIG. 2, which can be expressed as:

$\begin{matrix}G & R \\B & G\end{matrix}$

Multiple copies of this minimal repeating unit are tiled to cover theentire array of pixels in an image sensor. The minimal repeating unit isshown with a green pixel in the upper left corner, but three alternativeminimal repeating units can easily be discerned by moving the heavyoutlined area one pixel to the right, one pixel down, or one pixeldiagonally to the right and down. Although pixel block 102 is arepeating unit, it is not a minimal repeating unit because pixel block100 is a repeating unit and block 100 has fewer pixels than block 102.

An image captured using an image sensor having a two-dimensional arraywith the CFA of FIG. 2 has only one color value at each pixel. In orderto produce a full color image, there are a number of techniques forinferring or interpolating the missing colors at each pixel. These CFAinterpolation techniques are well known in the art and reference is madeto the following patents: U.S. Pat. Nos. 5,506,619; 5,629,734; and5,652,621.

Each pixel of image sensor 20 has both photodetector and activetransistor circuitry for readout of the pixel signal. The photo-detectorfor each pixel in the image sensor array converts photons impinging onthe pixel to an electric charge by the photoelectric effect. The chargeis integrated over a period of time that is long enough to collect adetectable amount of charge but short enough to avoid saturating storageelements. This integration time period is analogous to a film exposuretime (that is, shutter speed).

The timing of image capture can follow one of two basic patterns. In aglobal capture sequence, all image pixels are simply read at the sametime.

However, this type of sequence requires considerable device complexityand can be disadvantageous because it constrains the amount of space onthe sensor chip for photo-receptivity. Instead, a row-by-row readingmethod has been adopted and is often the preferred mode of reading forCMOS APS pixels.

In the image sensor array of a CMOS APS device, the integration time isthe time between a reset of a given row and a subsequent read of therow. Since only one row can be selected at a time, the reset/readroutine is sequential (i.e. row by row). This reading technique isreferred to as a “rolling electronic shutter” or, more simply, “rollingshutter” mode and is well known in the imaging art. A few examples ofvariations on rolling shutter time sequencing are given in U.S. Pat. No.6,115,065 entitled “Image Sensor Producing at Least Two IntegrationTimes from Each Sensing Pixel” to Yadid-Pecht et al. and in U.S. Pat.No. 6,809,766 entitled “Look-Ahead Rolling Shutter System in CMOSSensors” to Krymski et al. The shutter width for the read sequence isthe time between integration enable and readout. This can be of variablesize depending on the number of adjacent pixels that have the sameintegration time. The shutter width, having one or more rows read at atime, can also be adjusted by a fixed value to control the gain of anexposed area of a sensor array. As one method for rolling shuttersequencing, a reset pointer is indexed ahead of a read pointer by anamount equal to the shutter width. The time difference between the twopointers corresponds to the pixel integration time. As described above,the shutter width is completely analogous to the width of the physicalopening between the two curtains of a mechanical focal plane shutter. Inthe following, the term exposure duration will be used to correspond tothe integration time.

FIG. 3A shows a timing sequence for the rolling shutter mode asconventionally used under conditions of relatively good illumination.The abscissa (x-axis) represents time. The ordinate (y-axis) representsrows being read out of the image sensor. Each solid slanted line 302represents reading out all the rows of the image sensor in sequencestarting with the highest numbered rows and proceeding to the lowestnumbered rows. (Alternatively, the lines representing readout can beslanted upward from left to right to indicate reading out the rows fromlowest numbered rows to highest.) Each dashed line 300 representsresetting all the rows of the image sensor in sequence, again startingwith the highest numbered rows and proceeding to the lowest numberedrows, with the entire reset process requiring exactly as much time asthe readout process. The delay between a reset process 300 and itsimmediately following readout process 302 is the integration time forthe pixels 320, as indicated by the solid arrows. Note that theintegration time is constant for every row read out, but the integrationperiod for each row is time shifted with respect to the preceding andfollowing rows.

As can be seen from the timing diagram of FIG. 3A, this simple rollingshutter sequence permits periods during which no photons are obtained,specifically, between a read 302 and its immediately following reset300. Although this can be acceptable under good illumination, thisarrangement cannot perform well under low-light conditions. This isbecause more pixel integration time can be needed as light intensitydecreases. The timing diagram of FIG. 3B shows a timing sequence forlow-light conditions wherein the reset 300 is performed immediatelyfollowing or coincident with read 302. As a result, the pixelintegration time 320 has been increased to fill the time betweensuccessive reads and very few photons are wasted.

Even with the adoption of rolling shutter techniques, however, the taskof reading the image sensor efficiently still has its shortcomings Shearmotion artifacts are one type of problem. Relative motion between thescene (or elements of the scene) and the image sensor causes objectswithin the scene to appear distorted in the image captured by the imagesensor. This effect, termed image “shear”, is characteristic of rollingshutter arrangements. For example, if such a so-called rolling shutteror electronic focal plane shutter image sensor is used to capture animage of a car moving horizontally, the car moves relative to the imagesensor as each row of the captured image is exposed and read out, soeach row of the captured image shows the car at a different position.This can cause the round tires of the car to appear oval, and the car'srectangular windows to appear to be parallelograms. This distortion is adirect consequence of the amount of time required to read out all therows of the image sensor. Low-light performance can still be improvedand image dynamic range can still be less than what is desired.

One type of solution that has been proposed is the use of some portionof sensor array pixels as panchromatic pixels. For example, commonlyassigned U.S. Patent Application No. 2007/0024931 entitled “Image Sensorwith Improved Light Sensitivity” by Compton et al. discloses an imagesensor having both color and panchromatic pixels. In the context of thepresent disclosure, the term panchromatic pixel refers to a pixel havinga generally panchromatic photo-response, with a wider spectralsensitivity than the narrower spectral sensitivities represented in theselected set of color photo-responses. That is, a panchromatic pixel canhave high sensitivity to light across the entire visible spectrum.Although the panchromatic pixels generally have a wider spectralsensitivity than the set of color photo-responses, each panchromaticpixel can also have an associated filter. Such filter can be either aneutral density filter or a color or bandwidth filter.

Referring to the graph of FIG. 4, there are shown the relative spectralsensitivities of pixels with red, green, and blue color filters in atypical camera application. The X-axis in FIG. 4 represents lightwavelength in nanometers, spanning wavelengths approximately from thenear ultraviolet to near infrared, and the Y-axis represents efficiency(normalized). In FIG. 4, curve 110 represents the spectral transmissioncharacteristic of a typical bandwidth filter used to block infrared andultraviolet light from reaching the image sensor. Such a filter isneeded because the color filters used for image sensors typically do notblock infrared light, hence the pixels can be unable to distinguishbetween infrared light and light that is within the pass bands of theirassociated color filters. The infrared blocking characteristic shown bycurve 110 thus prevents infrared light from corrupting the visible lightsignal. The spectral quantum efficiency, i.e. the proportion of incidentphotons that are captured and converted into a measurable electricalsignal, for a typical silicon sensor with red, green, and blue filtersapplied is multiplied by the spectral transmission characteristic of theinfrared blocking filter represented by curve 110 to produce thecombined system quantum efficiencies represented by curve 114 for red,curve 116 for green, and curve 118 for blue. It is understood from thesecurves that each color photo-response is sensitive to only a portion ofthe visible spectrum. By contrast, the photo-response of the samesilicon sensor that does not have color filters applied (but includingthe infrared blocking filter characteristic) is shown by curve 112; thisis an example of a panchromatic photo-response. By comparing the colorphoto-response curves 114, 116, and 118 to the panchromaticphoto-response curve 112, it is clear that the panchromaticphoto-response can be three to four times more sensitive to widespectrum light than any of the color photo-responses. In rolling shuttermode, image sequences are typically read as illustrated in FIGS. 3A and3B. The entire image sensor is read and this constitutes one image inthe image sequence. Subsequently, the entire image sensor is read again,and this constitutes the next image in the image sequence.Alternatively, as described in U.S. patent application Ser. No.11/780,523, the image sensor is partitioned into disjoint subsets, andthese subsets are read in some relative order.

FIG. 5 shows a timing sequence for the rolling shutter mode for apreferred embodiment of the present invention wherein an image sensorarray has at least two groups of pixels wherein the number of pixels ofany group has no less than one-quarter of the number of pixels of theportion of the entire sensor that produces the digital image and whereinthe pixels of each group are uniformly distributed over the sensor. Therolling shutter read time given by 404 represents a readout of pixelsignals 422 of all of the pixels of the image sensor wherein a firstgroup of pixels (X1) and a second group of pixels (X2) can be read outindividually or binned. The pixels can be reset 404 after they are read.The X1 pixels are readout and reset according to the rolling shutterread time given by 402. The overall exposure for each X1 pixel is shownby line 410, which represents the time between the reset and read of theX1 pixels for each row of the image sensor. Pixel signals 420 representa read out of all of the X1 pixels. The read pattern repeats at thisstage, as the next readout is of the entire image sensor according tothe rolling shutter read time given by 404. The X1 pixels have been readout more than once. Some of the pixels readouts at 404 have a shorterexposure given by line 412, while other pixels have a longer exposuregiven by line 414. As a result, every photon that can reach the sensorcan be utilized.

Those skilled in the art will recognize that there are many alternativemethods to the present invention. The pixels can be binned or readoutindividually. More than two groups of pixels can be used to havemultiple exposure times. There can be a delay between the readoutprocess and the reset process for any group of pixels. The followingpreferred embodiments will detail some of these alternative methods.

FIG. 6 is a high-level flow chart of a preferred embodiment of thepresent invention. Initially a digital camera such as described in FIG.1 initiates a sequence of image captures 502 by exposing the imagesensor array 20 to scene light. The image sensor stage 28 in FIG. 1controls the process of selecting and reading pixels. The first timepixel signals are read from the sensor 504 through the analog signalprocessor 22, they produce a first set of pixel signals 510. The firstset of pixel signals 510 can be from the first group of pixels, or theycan be from the first and second group of pixels. If the first read isthe first group of pixels, all of the first groups of pixels are readfrom the sensor 506. If the first read is the first and second group ofpixels, all of the second group of pixels is read and the first group ofpixels are read from the image sensor 508. The image sensor continues tobe exposed until the capture process is terminated. The read processcontinues in a similar manner to the first read for subsequent readoutsfrom the sensor. For example, the second time pixel signals are readfrom the sensor, they produce a second set of pixel signals 510. Thesecond set of pixel signals 510 can be from the first group of pixels orfrom the first and second group of pixels. If the second read is fromthe first group of pixels, all first group of pixels are read again fromthe sensor 506. If the second read is from the first and second group ofpixels, all of the second group of pixels are read and the first groupof pixels are read again from the image sensor 508.

After the pixel signals 510 are read from the sensor, an imageprocessing step 512 operates on available pixel signals to generate adigital image 514. This process can be performed by the digital signalprocessor 36 in FIG.1. The image processing step 512 can utilize pixelsignals from the current readout as well as data buffered in memory 32from previous sensor readouts. The digital image 514 generated cancorrespond to the current sensor readout, or a previous sensor readout,or any combination of current and previous sensor readouts.

Relative to the pixel signals obtained from a given sensor readout, thedigital image 514 can have greater spatial resolution, improved imagequality, improved color information, or other enhancements. The sequenceof digital images can also have greater temporal resolution thanachievable simply by reading the sensor at the target spatialresolution. The digital image 514 is passed to a digital imageutilization function 516. This function can represent an encodingprocess, by which the digital image 514 is encoded into a videobitstream. It can represent a display function by which the digitalimage 514 is produced on a display. It can represent a printingfunction, by which the digital image 514 is printed. It can alsorepresent a sharing function by which the digital image 514 is sharedwith other devices. The aforementioned are examples of how the digitalimage 514 can be utilized, and are not limiting.

If the sequence of image captures is complete 518, the capture processis terminated 520. Otherwise, the capture process assesses whether thenext read from the sensor 522 is first group of pixels or first andsecond group of pixels and the readout and processing loop is iterateduntil the capture process is completed.

The proposed invention is capable of being used with an image sensorwith any color filter array pattern. The proposed invention is alsocapable of being used with an image sensor using only panchromaticpixels. In a preferred embodiment, however, the image sensor has bothpanchromatic pixels and color pixels. The panchromatic pixels are thefirst group of pixels and color pixels are the second group of pixels.In the proposed invention, an image sequence is captured by alternatelyreading all panchromatic pixels and reading color pixels and readingagain the panchromatic pixels.

The method of the present invention is described with respect to thecolor filter array pattern illustrated in FIG. 7. FIG. 7 illustrates anexample color filter array pattern 602 for a preferred embodiment of thepresent invention. In this example, approximately one-half of the pixelsare panchromatic 604, while the other one-half are color pixels splitamong red (R) 606, green (G) 608 and blue (B) 610. The color filterarray pattern 602 has a minimal repeating unit containing 16 pixels inthe following 4 by 4 array:

$\begin{matrix}P & G & P & R \\G & P & R & P \\P & B & P & G \\B & P & G & {P.}\end{matrix}$Those skilled in the art will recognize that other color filter arrayconfigurations and minimal repeating units are possible within the scopeof the present invention.

Pixels can be combined and these combined pixels can be read out. In apreferred embodiment, pixel combination is achieved through pixelbinning Various pixel binning schemes can be used during readout of theimage sensor, as illustrated in FIG. 8. In FIG. 8, two partial rows 701,702 of an image sensor are displayed. In this example, the underlyingreadout circuitry for a sensor array uses a floating diffusion 704 thatis switchably connected to one or more surrounding pixels at a time.Implementation and use of the floating diffusion is well known to thoseskilled in the digital image acquisition art. FIG. 8 shows aconventional arrangement in which each floating diffusion 704 servesfour surrounding pixels, shown in one example as a quartet 706.

Pixel signals can be switched to floating diffusion 704 in any of anumber of combinations. In a readout combination 708, each pixel inquartet 706 has its charge transferred separately to floating diffusion704 and thus is read individually. In a readout combination 710,panchromatic pixels P are binned, that is, share floating diffusion 704by emptying their stored charge to floating diffusion 704 at the sametime; similarly, both color (G) pixels in the quartet are binned,switching their signals at the same time to floating diffusion 704. Inthis binning scheme, panchromatic pixels are only combined with otherpanchromatic pixels, and color pixels are only combined with other colorpixels. In another readout combination 712, panchromatic pixels P arenot binned, but are read separately; here color pixels (G) are binned.In another readout combination 714, all four pixels connected to thegiven floating diffusion unit are binned simultaneously. In this binningscheme, panchromatic pixels are combined with both color pixels andother panchromatic pixels. Color pixels are combined with bothpanchromatic pixels and other color pixels.

FIG. 9 illustrates the readout pattern for an image sensor according toa preferred embodiment of the present invention. This figure is based onan image sensor with the color filter array pattern shown in FIG. 7, andutilizing a rolling shutter readout. In FIG. 9, several consecutivereadouts of pixel signals are shown. This collection of pixel signalsrepresents a portion of an overall image capture process. The entireimage capture process can contain additional readouts of pixel signalsextending in either direction along the time axis. Readout 804represents a readout of all of the pixels of the image sensorcorresponding to a combination of color pixels and panchromatic pixels.For each quartet of pixels connected to a floating diffusion unit, thetwo color pixels are binned and read as a single value. The twopanchromatic pixels are also binned separately and read as a singlevalue. Combined they give pixel signals 810 corresponding to a binnedpanchromatic/binned color readout. The pixels can be reset 804 afterthey are read. Each floating diffusion unit is accessed twice during thecourse of the sensor readout.

The color pixels are also reset according to rolling shutter reset timegiven by 806. The panchromatic pixels are read out and reset accordingto the rolling shutter time given by 802. The overall exposure for eachpanchromatic pixel is shown by the line 814, which represents the timebetween the reset and read of the panchromatic pixels for each row ofthe image sensor. The panchromatic pixels are readout without anybinning to produce pixel signals 812, such that the each floatingdiffusion unit is accessed twice during the overall readout of thepanchromatic pixels. Thus the readout of panchromatic pixels 802 and thereadout of a combination of panchromatic pixels and color pixels 804have the same readout rate and same motion shear properties. This designis advantageous not just for motion shear, but also for minimizingunused light while maintaining equal exposure duration for allpanchromatic pixels, as well as equal exposure duration for all colorpixels.

The panchromatic pixels are again reset according to the rolling shutterreset time given by 808. The read pattern repeats at this stage, as thenext readout is of the entire image sensor according to the rollingshutter read time given by 804. The panchromatic pixels readout at 804has a shorter exposure given by 816, while the color pixels have alonger exposure given by 818. Techniques for combining panchromaticpixels with relatively shorter exposure and color pixels with relativelylonger exposure are described by U.S. patent application Ser. No.12/258,389, filed Oct. 25, 2008 which is included herein by reference.

In FIG. 9, the panchromatic pixels are exposed for a different durationprior to an unbinned readout as they are prior to a binned readout. Inparticular, the panchromatic pixels are exposed for approximately halfas long prior to a binned readout as they are prior to an unbinnedreadout. This is an advantageous feature of the present invention, asthe panchromatic pixel signals are exposure balanced, such that thecharge read from the floating diffusion unit corresponding topanchromatic pixels is approximately equal whether binned at a shorterexposure duration or unbinned at a longer exposure duration. In FIG. 9,the exposure of the panchromatic pixels prior to a panchromatic pixelreadout partially overlaps the exposure of the color pixels prior to thefollowing readout of the color pixels.

FIG. 10 describes in greater detail one method of image processing 512after each readout of pixel signals is completed. Pixel signals that area combination of binned color pixels and binned panchromatic pixels 810can be color-interpolated from this initial color filter array data togenerate quarter-resolution color images 902. Panchromatic pixel signals812 can be spatially interpolated to generate sensor-resolutionpanchromatic images 904. A digital image 906 at the spatial resolutionof the image sensor is generated corresponding to each readout.Corresponding to readouts of panchromatic pixel signals 812, asensor-resolution, digital image 906 is generated using data from thesensor-resolution panchromatic image 904 as well as quarter-resolutioncolor images 902 from the previous and subsequent readouts ofpanchromatic and color pixel signals 810. Corresponding to readouts thatare a combination of panchromatic and color pixel signals 810, asensor-resolution, digital image 906 is generated using data from thequarter-resolution color image 902 as well as sensor-resolutionpanchromatic images 904 from the previous and subsequent readouts ofpanchromatic pixel signals 812. This scheme requires some buffering.Three readouts are used in the formation of each digital image 906.

The proposed invention allows the generation of an image sequence ofdigital images 906 with high spatial resolution, high temporalresolution, and high image quality. For image sequence capture accordingto prior methods, an image sequence of high spatial resolution isgenerated by reading the entire sensor repeatedly. The time required toreadout the full sensor is longer than the time required to readouteither binned panchromatic and binned color pixel signals 810 orpanchromatic pixel signals 812. Thus the temporal resolution, that isthe frame rate, of such an image sequence is lower than achieved usingthe proposed invention.

FIG. 11 describes in greater detail another method of image processing512 after each readout of pixel signals is completed. Pixel signals 810that are a combination of binned color pixels and binned panchromaticpixels can be color-interpolated from this initial color filter arraydata to generate quarter-resolution color images 902. Panchromatic pixelsignals 812 can be spatially interpolated to generate sensor-resolutionpanchromatic images 904. A digital image 1002 at one quarter the spatialresolution of the sensor—one-half the horizontal resolution and one-halfthe vertical resolution—is generated corresponding to each readout.Corresponding to panchromatic pixel signals 812, an enhancedquarter-resolution digital image 1002 is generated using data from thesensor-resolution panchromatic image 904 as well as quarter-resolutioncolor images 902 from the previous and subsequent readouts ofpanchromatic and color pixel signals. Corresponding to readouts that area combination of panchromatic and color pixel signals 810, an enhancedquarter-resolution digital image 1002 is generated using data from thequarter-resolution color image 902 as well as sensor-resolutionpanchromatic images 904 from the previous and subsequent panchromaticpixel signals 812. This scheme requires some buffering. Three readoutsare used in the formation of each digital image 1002.

The proposed invention allows the generation of an image sequence ofdigital images 1002 with improved spatial resolution, high temporalresolution, and high image quality. For image sequence capture accordingto prior methods, an image sequence of quarter-resolution images can begenerated by binning or sub-sampling and reading the sensor repeatedly.In order to read out the image sequence at a high temporal resolution,each readout is binned or sub-sampled to one quarter the resolution ofthe sensor. Thus the spatial resolution of the image sequence islimited. In the proposed invention, readouts of panchromatic pixelsignals 812 have the spatial resolution of the sensor, and thus improvedhigh spatial frequency information can be maintained in the digitalimages 1002. Additionally, in the proposed method color pixels can haveexposure durations longer than the inverse of the frame rate. In imagesequence capture according to prior methods, this is not possible sinceeach readout is a full sensor readout. Extended color pixel exposureduration allows improved signal to noise ratios to be obtained for colorpixels and improves the overall image quality of the digital images.

FIG. 12 describes in greater detail another method of image processing512 after each readout of pixel signals is completed. Pixel signals thatare a combination of binned color pixels and binned panchromatic pixels810 can be color-interpolated from this initial color filter array datato generate quarter-resolution color images 902. Panchromatic pixelsignals 812 can be spatially interpolated to generate sensor-resolutionpanchromatic images 904. Corresponding to panchromatic pixel signals812, both an enhanced quarter-resolution (one-half horizontal spatialresolution, one-half vertical spatial resolution), digital image 1002 aswell as an enhanced sensor-resolution digital image 906 are generatedusing data from the sensor-resolution panchromatic image 904 as well asquarter-resolution color images 902 from the previous and subsequentreadouts of panchromatic and color pixel signals 810. Corresponding toreadouts of panchromatic and color pixel signals 810, an enhancedquarter-resolution digital image 1002 is generated using data from thequarter-resolution color image 902 as well as sensor-resolutionpanchromatic images 904 from the previous and subsequent panchromaticpixel signals 812. This scheme requires some buffering. Three readoutsare used in the formation of each quarter-resolution digital image 906or sensor-resolution digital image 1002.

The proposed invention allows the simultaneous generation of a lowspatial resolution, high frame rate image sequence as well as a highspatial resolution, low frame rate image sequence. Thus it is possibleto capture simultaneously both a low resolution image sequence as wellas a high resolution, high quality still image. Prior solutions forsimultaneously capturing an image sequence and a still image typicallyrequire additional hardware, or must disrupt the image sequence captureto acquire the still image.

For the image processing 512 described in FIG. 12, there are multipleoptions how to treat the digital images 514. In one method, thequarter-resolution color images 902 are treated as a first sequence, andthe sensor-resolution color images are treated as an independent secondsequence, and the two sequences are stored as separate sequences. FIG.13 illustrates an alternative method for handling the quarter-resolutionand sensor-resolution digital images 514. A quarter-resolution colorimage 1202 is up-sampled in an up-sampling block 1206 to produce asensor-resolution up-sampled color image 1208. A residual image 1212 isformed in a residual image formation block 1210 by subtracting thesensor-resolution up-sampled color image 1208 from the sensor-resolutioncolor image 1204. This residual image 1212 is stored as well as thequarter-resolution color image 1202. The two images can be storedseparately or combined. In one example, quarter-resolution color images1202 can be stored using a compression syntax and file format such asthe JPEG compression standard and TIFF file format, and the residualimage 1212 can be stored as metadata within the file. In anotherexample, quarter-resolution color images 1202 can be stored using acompression syntax and file format such as the MPEG compression standardand Quicktime.MOV file format, and the residual images 1212 can bestored as metadata within the file. File readers can ignore the metadataand just decode the quarter-resolution color images. Intelligent filereaders can extract the metadata as well and reconstruct thesensor-resolution color images.

FIG. 14 illustrates the readout pattern and image generation for animage sensor according to another preferred embodiment of the presentinvention. This figure is based on an image sensor with the color filterpattern shown in FIG. 7, and utilizing a rolling shutter readout. InFIG. 14, several consecutive readouts are shown. This collection ofpixel signals represents a portion of an overall image capture process.The entire image capture process can contain additional readouts ofpixel signals extending in either direction along the time axis. Readout1302 represents a readout of all of the pixels of the image sensor,corresponding to a combination of color pixels and panchromatic pixels.For each quartet of pixels connected to a floating diffusion unit, thetwo color pixels and two panchromatic pixels are binned and read as asingle value. Combined they produce pixel signals 1308 having a dilutedBayer pattern. The pixels are also reset at 1302 after they are read.Each floating diffusion unit is accessed once during the course of thesensor readout.

The color pixels are also reset according to rolling reset time given by1304. The panchromatic pixels are read and reset according to therolling shutter time given by 1306. The overall exposure for eachpanchromatic pixel is shown by the line 1316, which represents the timebetween the reset and readout of the panchromatic pixels for each row ofthe image sensor. The panchromatic pixel signals 1310 are generated byreading the panchromatic pixels with binning, such that the eachfloating diffusion unit is accessed once during the overall readout ofthe panchromatic pixels. Thus the panchromatic pixel readout 1306 andthe readout 1302 of panchromatic pixels and color pixels have the samereadout rate and same motion shear properties. This design isadvantageous not just for motion shear, but also for minimizing unusedlight while maintaining equal exposure duration for all panchromaticpixels, as well as equal exposure duration for all color pixels.

The read pattern repeats at this stage as the next readout is of theentire image sensor according to the rolling shutter reset time given by1302. The panchromatic pixels read at 1302 have an exposure given by1318, while the color pixels have an exposure given by 1320. FIG. 14shows the color pixels with a longer exposure duration 1320 thanexposure duration 1318 of the panchromatic pixels readout at pixelreadout 1302. This relationship is not fixed, however, and in fact thecolor pixel exposure duration 1320 can also be shorter than, or equal tothe exposure duration of the panchromatic pixels 1318.

FIG. 14 also describes in greater detail one method of image processing512 after each readout of pixel signals is completed. A readout ofbinned color and panchromatic pixel signals 1308 can be combined withneighboring readouts of binned panchromatic pixel signals 1310 togenerate an improved panchromatic and color pixel image 1312. Similarly,a readout of binned panchromatic pixel signals 1310 can be combined withneighboring readouts of binned color and panchromatic pixel signals 1308to generate an improved panchromatic and color pixel image 1312. Theimproved panchromatic and color pixel image 1312 can be processed toproduce an enhanced quarter-resolution color image 1314. This schemerequires some buffering. Three readouts are used in the formation ofeach digital image 1314.

Another embodiment of the present invention provides an extended dynamicrange image. This is described in more detail in FIG. 15 and FIG. 16with reference to FIG. 5. FIG. 15 provides an exemplary pixel arraypattern 1402. In this example, pixels (X1) 1404 represent a first groupof pixels and pixels (X2) 1406 represent a second group of pixels. X1and X2 represent panchromatic pixels. FIG. 5 shows a timing sequence forthe rolling shutter mode using the color filter array pattern shown inFIG. 15.

FIG. 16 describes in greater detail a method of image processing 512from FIG. 6 after each readout of pixel signals is completed withrespect to the pixel array pattern in FIG. 15. Pixel signals 420 can beappropriately processed from this initial array data to generate images430. Pixel signals 422 can be appropriately processed from this initialarray data to generate images 432. In this case, images 430 and 432 arehalf of the horizontal resolution of the sensor. The pixel signals foreach readout are scaled by the ratio of the longest exposure for all ofthe readouts and the exposure for the readout. The sum of the scaledpixels signals are normalized by the number of pixel signals of thereadout to produce the image values for the readout. Eq. 1 shows asimple method for computing the image values

$\begin{matrix}{{{Pe} = {\sum\limits_{r}{\left( {\sum\limits_{g}{\left( {{T_{f}/T_{r,g}}*{pixelssignals}_{r,g}} \right)/g}} \right)/r}}},} & {{EQ}.\mspace{14mu} 1}\end{matrix}$wherein Pe represents the extended dynamic range image values, rrepresents the readout, g represents the number of groups within thereadout, T_(f) represents the longest exposure, T_(r.g) represents theexposure for the readout and group, and pixel signals_(r.g) representsthe pixel signals for the readout and group. If a pixel signal is abovean upper threshold or below a lower threshold, then it is not used inthe calculation for the image value and the number of groups of thereadout (g) is adjusted accordingly.

The following description for FIG. 16 will demonstrate the calculationin a step-by-step process. Referring to FIG. 5, the exposure shown byline 414, corresponds to the longest exposure (TF1). The image values Pafor pixel signals 420, is given by EQ. 2Pa=(TF1/TS1)*X1(420)/g(420)   EQ 2,wherein X1 (420) was exposed with exposure (TS1) shown by line 410,frame exposure (TF1) is shown by line 414, and number of pixel signalsof the readout is g(420). The number of pixel signals of the readout, g,is the number of groups of pixels that are read for that particularreadout. Since the set of pixels signals 420 only contains one group ofpixels (X1), the value for g(420) is 1 in EQ.2. Similarly, the imagevalues Pb for pixel signals 422 is given by EQ. 3Pb=((TF1/TS2)*X1(422)+X2(422))/g(422)   EQ 3,wherein X1(422 ) was exposed with exposure (TS2) shown by line 412,X2(422) was exposed with longest exposure (TF1) shown by line 414, andnumber of pixel signals of the readout is g(422). The value for g(422)in EQ 2. is 2 because there are two groups of pixels used to compute Pb.If the value of X1 or X2 is above an upper threshold or below a lowerthreshold, then it is not used in the calculation for the image valueand the number of groups of the readout (g) is adjusted accordingly.Half-resolution images 432 and 430 are merged to produce a digital image434 with an extended dynamic range value Pe. In another example, Pa andPb are summed and divided by the number of half-resolution images toproduce a half resolution digital image. In another example, the valuePe is interpolated from Pa and Pb to produce the digital image. Thoseskilled in the art will recognize that there are many alternativemethods for calculating the extended dynamic range image.

FIG. 17 provides an exemplary pixel array pattern 1412 for anotherpreferred embodiment of the present invention. In this example, pixels(X1) 1414, (X2) 1416, (X3) 1418 and (X4) 1420 represent four differentgroups of panchromatic pixels. In another embodiment, one or more of thepanchromatic pixels can be replaced with a color pixel. The color pixelcan provide color information to the final processed image. The colorpixel can provide a different sensitivity than the panchromatic pixelsas was previously described in FIG. 4.

FIG. 18 illustrates the readout pattern and image generation for animage sensor according to another preferred embodiment of the presentinvention. This figure is based on an image sensor with the pixel arraypattern shown in FIG. 17, and utilizing a rolling shutter readout. Thiscollection of pixel signals represents a portion of an overall imagecapture process. The entire image capture process can contain additionalreadouts of pixel signals extending in either direction along the timeaxis. The rolling shutter read time given by 1504 represents a readoutof pixel signals 1510 of all of the pixels (X1, X2, X3 and X4) of theimage sensor. The pixels can be reset 1504 after they are read.

The X1 pixels are readout and reset according to the rolling shutterread time given by 1502. The overall exposure (TS1) for each X1 pixel isshown by the line 1512, which represents the time between the reset andread of the X1 pixels for each row of the image sensor. Pixel signals1520 represents a readout of all of the X1 pixels.

The X1 and X2 pixels are readout and reset again according to therolling shutter read time given by 1506. The overall exposure (TS2) foreach X1 pixel is shown by the line 1515, which represents the timebetween the reset and read of the X1 pixels for each row of the imagesensor. The overall exposure (TS3) for each X2 pixel is shown by theline 1516, which represents the time between the reset and read of theX2 pixels for each row of the image sensor. Pixel signals 1522represents a readout of all of the X1 and X2 pixels of the image sensor.This design is advantageous because it minimizes unused light whileextending the dynamic range of the sensor.

The read pattern repeats at this stage, as the next readout is of theentire image sensor according to the rolling shutter read time given by1504. Some of the pixels readout at 1504 have a shorter exposure (TS4)given by 1517, while other pixels have a longer exposure (TF1) given by1518.

In FIG. 18, the pixels are exposed for different durations. This is anadvantageous feature of the present invention, as the differentexposures allow for a greater dynamic range within a group of pixels.Another advantageous feature of the present invention is that everyphoton that can reach the sensor is read and available to be used. Inaddition, some pixel groups can be binned with other pixels groups toeffectively double the sensitivity. For example, pixels from group X3and X4 can be binned for pixel signals 1510. Pixels from group X1 and X2can be binned for pixel signals 1510. Pixel groups with equal exposurefor a given readout can be binned.

The image values calculation proceeds similar to the explanation givenfor FIG. 16. Pixel signals 1520 can be appropriately processed from thisinitial array data to generate images 430. Pixel signals 1522 can beappropriately processed from this initial array data to generate images432. Pixel signals 1510 can be appropriately processed from this initialarray data to generate images 436. Image values Pa are calculated fromEQ 2. Image values Pb are calculated from EQ 4Pb=((TF1/TS2)*X1+(TF1/TS3)*X2)/g   EQ 4,wherein X1(1522) was exposed with exposure (TS2) shown by line 1515,X2(1522) was exposed with exposure (TS3) shown by line 1516, longestexposure (TF1) is shown by line 1518, and number of pixel signals of thereadout is g(1522). The image values Pf for pixel signals 1510 are givenby EQ. 5Pf=((TF1/TS4)*(X1+X2)+X3+X4)/g   EQ 5,wherein X1(1510) and X2(1510) were exposed with exposure (TS3) shown byline 1517, X3(1510) and X4(1510) were exposed with longest exposure(TF1) shown by line 1518, and number of pixel signals of the readout isg(1510). If pixel groups X1 and X2 were binned for readout 1510, ascalar would be applied to the sum of X1 and X2 to accommodate for thebinning Similarly, a scalar would be applied to the sum of X3 and X4 ifthey were binned for readout 1510. Images 430, 432 and 436 are merged toproduce a digital image 434 with an extended dynamic range value Pe

Those skilled in the art will recognize that conventional automaticexposure techniques and circuitry can be adapted to accommodate themultiple sets of signals provided by embodiments of the presentinvention. The charge is integrated over a period of time that is longenough to collect a detectable amount of charge for all pixels whileless constraint is required to avoid saturating storage elements. Byoptimizing the exposure durations for each of the groups of pixels for agiven scene, normally clipped highlight regions of the image and darkregions can be properly exposed. A scene with a high dynamic range willprovide a higher dynamic range image than a scene with a lower dynamicrange.

FIG. 19 illustrates an example color filter array pattern 1702 for apreferred embodiment of the present invention. In this example, pixelsare panchromatic (P) 1704, red (R) 1706, green (G) 1708 and blue (B)1710. The color filter array pattern 1702 has a minimal repeating unitcontaining 16 pixels in the following 4 by 4 array:

$\begin{matrix}G & P & B & P \\P & P & P & P \\R & P & G & P \\P & P & P & {P.}\end{matrix}$

Those skilled in the art will recognize that other color filter arrayconfigurations and minimal repeating units are possible within the scopeof the present invention. FIG. 20 illustrates the readout pattern andimage generation for an image sensor according to another preferredembodiment of the present invention. This figure is based on an imagesensor with a color filter array pattern shown in FIG. 19, and utilizinga rolling shutter readout. This is a slight variation from the readoutpattern illustrated in FIG. 18 in that a color pixel has replaced one ofthe panchromatic pixels X2. This is useful in applications that requiregreater dynamic range with some color information. This collection ofpixel signals represents a portion of an overall image capture process.The entire image capture process can contain additional readouts ofpixel signals extending in either direction along the time axis. Therolling shutter read time given by 1804 represents a readout of pixelsignals 1810 of all of the pixels (X1, color, X3 and X4) of the imagesensor. The pixels can be reset 1804 after they are read.

The X1 pixels are readout and reset according to the rolling shutterread time given by 1802. The overall exposure for each X1 pixel is shownby the line 1812, which represents the time between the reset and readof the X1 pixels for each row of the image sensor. Pixel signals 1820represents a readout of all of the X1 pixels.

The X1 and X4 pixels are readout and reset again according to therolling shutter read time given by 1806. The overall exposure for eachX1 pixel is shown by the line 1815, which represents the time betweenthe reset and read of the X1 pixels for each row of the image sensor.The overall exposure for each X4 pixel is shown by the line 1816, whichrepresents the time between the reset and read of the X4 pixels for eachrow of the image sensor. Pixel signals 1822 represents a readout of allof the X1 and X4 pixels of the image sensor. This design is advantageousbecause it minimizes unused light while extending the dynamic range ofthe sensor.

The read pattern repeats at this stage, as the next readout is of theentire image sensor according to the rolling shutter read time given by1804. Some of the pixels readout at 1804 have a shorter exposure givenby 1817, while other pixels have a longer exposure given by 1818.

In FIG. 18, the pixels are exposed for different durations. This is anadvantageous feature of the present invention, as the differentexposures allow for a greater dynamic range within a group of pixels.Another advantageous feature of the present invention is that everyphoton that can reach the sensor is read and available to be used. Inaddition, some pixel groups can be binned with other pixels groups toeffectively double the sensitivity. For example, pixels from group X3and color can be binned for pixel signals 1810. Pixels from group X1 andX4 can be binned for pixel signals 1810. Pixel groups with equalexposure for a given readout can be binned.

Those skilled in the art will recognize that there are many alternativemethods to the present invention.

The invention has been described in detail with particular reference tocertain preferred embodiments thereof, but it will be understood thatvariations and modifications can be effected within the scope of theinvention as described above, and as noted in the appended claims, by aperson of ordinary skill in the art without departing from the scope ofthe invention.

PARTS LIST

-   10 Light-   11 Imaging stage-   12 Lens-   13 Filter block-   14 Iris-   16 Sensor block-   18 Shutter block-   20 Image Sensor-   22 Analog signal processor-   24 A/D converter-   26 Timing generator-   28 Sensor stage-   30 Bus-   32 DSP Memory-   36 Digital signal processor-   38 Processing Stage-   40 Exposure controller-   50 System controller-   52 Bus-   54 Program memory-   56 System memory-   57 Host interface-   60 Memory card interface-   62 Socket-   64 Memory card-   68 User interface-   70 Viewfinder display-   72 Exposure display-   74 User inputs-   76 Status display-   80 Video encoder-   82 Display controller-   88 Image display-   100 Block-   102 Block-   110 Filter transmission curve-   112 Panchromatic photo-response curve-   114 Color photo-response curve-   116 Color photo-response curve-   118 Color photo-response Curve-   300 Reset process-   302 Readout process-   320 Pixel integration time-   402 Rolling shutter read time-   404 Rolling shutter read time-   410 Pixel exposure-   412 Pixel exposure-   414 Pixel exposure-   420 Pixel signals-   422 Pixel signals-   430 Image-   432 Image-   434 Digital Image-   436 Image-   502 Sequence of image capture-   504 Sensor-   506 Sensor-   508 Image Sensor-   510 Pixel signals-   512 Image processing step-   514 digital image-   516 digital image utilization-   518 Image capture complete query-   520 Image capture process termination-   522 Next read from sensor-   602 Color filter array pattern-   604 Panchromatic pixel-   606 Red pixel-   608 Green pixel-   610 Blue pixel-   701 Row-   702 Row-   704 Floating diffusion-   706 Pixel quartet-   708 Readout combination-   710 Readout combination-   712 Readout combination-   714 Readout combination-   802 Panchromatic pixel readout-   804 Pixel readout and reset-   806 Rolling shutter reset time-   808 Rolling shutter reset time-   810 Panchromatic and color pixel signals-   812 Pixel signals-   814 Panchromatic pixel exposure-   816 Panchromatic pixel exposure-   818 Color pixel exposure-   902 Quarter-resolution color image-   904 Sensor-resolution panchromatic image-   906 digital image-   1002 digital image-   1202 Quarter-resolution color image-   1204 Sensor-resolution color image-   1206 Up-sample block-   1208 Sensor-resolution up-sampled color image-   1210 Residual image formation block-   1212 Residual image-   1302 Pixel readout and reset-   1304 Color pixel reset-   1306 Panchromatic pixel readout and reset-   1308 Pixel signals-   1310 Panchromatic pixel signals-   1312 Color pixel image-   1314 digital image-   1316 Panchromatic pixel exposure-   1318 Panchromatic pixel exposure-   1320 Color pixel exposure-   1402 Pixel array pattern-   1404 First group of pixels-   1406 Second group of pixels-   1412 Pixel array pattern-   1414 First group of pixels-   1416 Second group of pixels-   1418 Third group of pixels-   1420 Fourth group of pixels-   1502 Rolling shutter read time-   1504 Rolling Shutter Read time-   1506 Rolling shutter read time-   1510 Pixel signals-   1512 Pixel exposure-   1515 Pixel exposure-   1516 Pixel exposure-   1517 Pixel exposure-   1518 Pixel exposure-   1520 Pixel signals-   1522 Pixel signals-   1702 Color filter array pattern-   1704 First group of pixels-   1706 Second group of pixels-   1708 Third group of pixels-   1710 Fourth group of pixels-   1802 Pixel readout and reset-   1804 Pixel readout and reset-   1806 Pixel readout and reset-   1810 Pixel signals-   1812 Pixel exposure-   1815 Pixel exposure-   1816 Pixel exposure-   1817 Pixel exposure-   1818 Pixel exposure-   1820 Pixel signals-   1822 Pixel signals

1. A method for producing a digital image, the method comprising:providing an image sensor with at least a first group of pixels and asecond group of pixels; reading out first pixel signals from the firstgroup of pixels, the first pixel signals generated during a firstexposure duration, wherein the first group of pixels is reset afterreading out the first pixel signals generated during the first exposureperiod; reading out second pixels signals from the first group of pixelsafter resetting the first group of pixels, the second pixel signalsgenerated during a second exposure period; reading out third pixelsignals from the second group of pixels, the third pixel signalsgenerated during a third exposure period that overlaps at least aportion of the first and second exposure periods; and using at least thefirst, second, and third pixel signals to produce the digital image. 2.The method of claim 1, wherein the second pixel group includes at leastone color pixels and the first pixel group includes at least onepanchromatic pixels.
 3. The method of claim 1 further comprising:reading out fourth pixel signals from the first group of pixels afterreading out the second pixel signals and then resetting the first pixelgroup, the fourth pixel signals generated during a fourth exposureperiod following the second exposure period; reading out fifth pixelsignals from a third group of pixels, the fifth pixel signals generatedduring a fifth exposure period that overlaps at least a portion of thefirst, second, and third exposure periods, wherein the image sensorincludes the first, second, and third groups of pixels; and using thefourth and fifth pixel signals to produce the digital image.
 4. Themethod of claim 3 further comprising: reading out sixth pixel signalsfrom the second group of pixels after reading out the third pixelsignals and then resetting the second pixel group, the sixth pixelsignals generated during a sixth exposure period following the thirdexposure period; and using the sixth pixel signals to produce thedigital image.
 5. The method of claim 4 further comprising: reading outseventh pixel signals from a fourth pixel group, the seventh pixelsignals generated during a seventh exposure period substantially thesame as the fifth exposure period; and using the seventh pixel signalsto produce the digital image.
 6. The method of claim 5, wherein thesecond pixel group includes at least one color pixels, and wherein thefirst, third, and fourth pixel groups include at least one panchromaticpixels.
 7. The method of claim 5, wherein the second pixel group is acolor pixel and the first, third, and fourth pixel groups arepanchromatic (P) pixels.
 8. The method of claim 7, wherein a first,second, third, and fourth pixel configuration of a pixel pattern eachincludes the first, second, third, and fourth pixel groups, wherein thecolor pixel of the first, second, third, and fourth pixel module is red(R), green (G), green (G), and blue (B), respectively.
 9. The method ofclaim 8, wherein the pixel pattern is arranged in the image sensor as:$\begin{matrix}G & P & B & P \\P & P & P & P \\R & P & G & P \\P & P & P & {P.}\end{matrix}$
 10. The method of claim 5, wherein the first (1), second(2), third (3), and fourth (4) pixel groups are arranged in a pixelpattern in the image sensor as: $\begin{matrix}12 \\34.\end{matrix}$
 11. The method of claim 5, wherein the fourth pixelsignals and the sixth pixel signals are binned.
 12. The method of claim5, wherein the fifth pixel signals and the seventh pixel signals arebinned.
 13. The method of claim 1, wherein the first group of pixels (1)and the second group of pixels (2) are panchromatic pixels arranged in apixel pattern as: $\begin{matrix}12 \\{21,}\end{matrix}$ and wherein producing the digital image includes:processing the first pixel signals to generate a first image andprocessing the second and third pixel signals to generate a secondimage, wherein the first image and the third image are half of thehorizontal resolution of the image sensor; and combining the first imageand the second image to generate the digital image with extended dynamicrange.
 14. The method of claim 13, wherein processing the first pixelsignals to generate a first image and processing the second and thirdpixel signals to generate the second image includes: scaling the first,second, and third pixel signals by a ratio to generate first, second,and third scaled pixel signals, the ratio being a first duration of thethird exposure period divided by an actual duration corresponding to therespective exposure period of the respective pixel signals; andnormalizing a sum of the first, second, and third scaled pixel signalsby a number of pixel signals of each respective readout.
 15. An imagingsystem comprising: a pixel array including at least a first group ofpixels and a second group of pixels; control logic coupled to acquireimage data from the pixel array; and a non-transistorymachine-accessible storage medium that provides instructions that, whenexecuted by the imaging system, will cause the imaging system to performoperations comprising: reading out first pixel signals from the firstgroup of pixels, the first pixel signals generated during a firstexposure duration, wherein the first group of pixels is reset afterreading out the first pixel signals generated during the first exposureperiod; reading out second pixels signals from the first group of pixelsafter resetting the first group of pixels, the second pixel signalsgenerated during a second exposure period; reading out third pixelsignals from the second group of pixels, the third pixel signalsgenerated during a third exposure period that overlaps at least aportion of the first and second exposure periods; and using at least thefirst, second, and third pixel signals to produce a digital image. 16.The imaging system of 15, wherein the second pixel group includes atleast one color pixels and the first pixel group includes at least onepanchromatic pixels.
 17. The imaging system of 15, with furtherinstructions stored in the non-transitory machine accessible storagemedium, that when executed by the imaging system, will cause the imagingsystem to perform operations comprising: reading out fourth pixelsignals from the first group of pixels after reading out the secondpixel signals and then resetting the first pixel group, the fourth pixelsignals generated during a fourth exposure period following the secondexposure period; reading out fifth pixel signals from a third group ofpixels, the fifth pixel signals generated during a fifth exposure periodthat overlaps at least a portion of the first, second, and thirdexposure periods, wherein the pixel array includes the first, second,and third groups of pixels; and using the fourth and fifth pixel signalsto produce the digital image.
 18. The imaging system of 17 with furtherinstructions stored in the non-transitory machine accessible storagemedium, that when executed by the imaging system, will cause the imagingsystem to perform operations comprising: reading out sixth pixel signalsfrom the second group of pixels after reading out the third pixelsignals and then resetting the second pixel group, the sixth pixelsignals generated during a sixth exposure period following the thirdexposure period; and using the sixth pixel signals to produce thedigital image.
 19. The imaging system of 18 with further instructionsstored in the non-transitory machine accessible storage medium, thatwhen executed by the imaging system, will cause the imaging system toperform operations comprising: reading out seventh pixel signals from afourth pixel group, the seventh pixel signals generated during a seventhexposure period substantially the same as the fifth exposure period; andusing the seventh pixel signals to produce the digital image.
 20. Theimaging system of 19, wherein the second pixel group includes at leastone color pixels, and wherein the first, third, and fourth pixel groupsinclude at least one panchromatic pixels.
 21. The imaging system of 19,wherein the second pixel group is a color pixel and the first, third,and fourth pixel groups are panchromatic (P) pixels.
 22. The imagingsystem of 21, wherein a first, second, third, and fourth pixelconfiguration of a pixel pattern each includes the first, second, third,and fourth pixel groups, wherein the color pixel of the first, second,third, and fourth pixel module is red (R), green (G), green (G), andblue (B), respectively.
 23. The imaging system of 22, wherein the pixelpattern is arranged in the pixel array as: $\begin{matrix}G & P & B & P \\P & P & P & P \\R & P & G & P \\P & P & P & {P.}\end{matrix}$
 24. The imaging system of 19, wherein the first (1),second (2), third (3), and fourth (4) pixel groups are arranged in apixel pattern in the pixel array as: $\begin{matrix}12 \\34.\end{matrix}$
 25. The imaging system of 19, wherein the fourth pixelsignals and the sixth pixel signals are binned.
 26. The imaging systemof 19, wherein the fifth pixel signals and the seventh pixel signals arebinned.
 27. The imaging system of claim 15, wherein the first group ofpixels (1) and the second group of pixels (2) are panchromatic pixelsarranged in a pixel pattern as: $\begin{matrix}12 \\{21,}\end{matrix}$ and wherein producing the digital image includes:processing the first pixel signals to generate a first image andprocessing the second and third pixel signals to generate a secondimage, wherein the first image and the third image are half of thehorizontal resolution of the image sensor; and combining the first imageand the second image to generate the digital image with extended dynamicrange.
 28. The imaging system of claim 27, wherein processing the firstpixel signals to generate a first image and processing the second andthird pixel signals to generate the second image includes: scaling thefirst, second, and third pixel signals by a ratio to generate first,second, and third scaled pixel signals, the ratio being a first durationof the third exposure period divided by an actual duration correspondingto the respective exposure period of the respective pixel signals; andnormalizing a sum of the first, second, and third scaled pixel signalsby a number of pixel signals of each respective readout.
 29. A methodfor producing a digital image, the method comprising: providing an imagesensor with two color pixels and two panchromatic pixels, wherein thetwo color pixels and the two panchromatic share a floating diffusion;resetting the two panchromatic pixels; reading out unbinned panchromaticsignals of the two panchromatic pixels generated during a first exposureperiod initiated after resetting the two panchromatic pixels; readingout binned color signals of the two color pixels generated during asecond exposure period initiated before reading out the unbinnedpanchromatic signals; reading out binned panchromatic signals of the twopanchromatic pixels generated during a third exposure period initiatedafter reading out the unbinned panchromatic signals; and using at leastthe unbinned panchromatic signals, the binned color signals, and thebinned panchromatic signals to produce the digital image.
 30. The methodof claim 29, wherein reading out the binned color signals and the binnedpanchromatic signals is during a same shutter.
 31. The method of claim30, wherein the same shutter is a rolling shutter.
 32. The method ofclaim 29, wherein a first duration of the first exposure period isapproximately two times a second duration of the third exposure period.33. The method of claim 32, wherein a third duration of the secondexposure period is approximately equal to a sum of the first durationand the second duration.
 34. The method of claim 29, wherein producingthe digital image includes color-interpolating the binned color signalsand the binned panchromatic signals and spatially interpolating theunbinned panchromatic signals.
 35. The method of claim 34, wherein thedigital image has the spatial resolution of the image sensor.
 36. Themethod of claim 34, wherein the digital image is an enhancedquarter-resolution digital image that is one half the verticalresolution of the image sensor and one half the horizontal resolution ofthe image sensor.
 37. A method for producing a digital image, the methodcomprising: providing an image sensor having pairs of color pixelssharing floating diffusions with pairs of panchromatic pixels, the pairsof color pixels including a pair of first color pixels, a pair of secondcolor pixels, and a pair of third color pixels; resetting each of thepairs of panchromatic pixels and initiating a first exposure period;reading out binned panchromatic signals from the pairs of panchromaticpixels, wherein the panchromatic signals were generated during the firstexposure period; resetting each of the pairs of panchromatic pixels andinitiating a second exposure period after reading out the binnedpanchromatic signals; resetting each of the pairs of color pixels andinitiating a third exposure period; reading out binnedcolor/panchromatic signals, wherein the binned color/panchromaticsignals are generated during the second and third exposure period andthe binned color/panchromatic signals include image charge from thepairs of color pixels and the pairs of panchromatic pixels storedtogether in each of the floating diffusions; and generating the digitalimage using at least the binned panchromatic signals and the binnedcolor/panchromatic signals.
 38. The method of claim 37, wherein a firstduration of the first exposure period is approximately equal to a secondduration of the second exposure period.
 39. The method of claim 37,wherein the third exposure period starts before the second exposureperiod.
 40. The method of claim 37, wherein the third exposure periodstarts after the second exposure period.
 41. The method of claim 37,wherein the first color pixels are red pixels, the second color pixelsare green pixels, and the third color pixels are blue pixels.
 42. Themethod of claim 41, wherein the binned color/panchromatic signalsinclude a diluted Bayer pattern.